Models of Mental Workload 3 Subjective Measures of Mental Workload

The primary objective of this research project was to investigate models for monitoring and predicting subjective workload in the control of complex systems. Such models would enable systems to use workload levels to distribute tasks optimally in addition to identifying levels of workload which could lead to a serious breakdown in performance. In the aircraft-pilot system, for example, such capabilities could provide warnings to the pilot of high workload levels and could also assess ways of reducing the pilot’s workload by offering to assume control of some ongoing tasks. In this initial project, we tried to determine how well a model can assess workload using information about task requirements and task performance. Participants rated subjective workload levels after each block of trials. The blocks consisted of various combinations of three tasks with varying levels of difficulty. The workload ratings and the performance data were used to create a database for developing models. The tasks were: (a) a continuous tracking task with a random forcing function and three different updating speeds; (b) a discrete tracking task in which response keys were pressed to indicate the position of a target in one of four different locations; and (c) a tone-counting task which required counting the number of higher pitched tones in a series of tones of 800 or 1200 Hz. Neural net models applied to group data consisting of eight individuals were able to achieve 85-95% accuracy in predicting a “redline” workload level in training data. On completely new data, accuracy was in the 70-75% range. The redline value was adopted from earlier work (Reid & Colle, 1988) showing that at that value of workload, performance measures begin to show effects of workload. Prediction in the 70-75% range is of interest theoretically, but for practical utility, values in the range of 90-95% are desirable. When we developed models from the data of individual subjects, such levels were reached for two of eight subjects, but the other six were lower. The average accuracy from individual subject models was better than that obtained with group data suggesting that individual models are a more promising direction to pursue. We conclude with a recommendation to add physiological measures collected during the performance of the tasks to assist in predicting workload. Given the levels of accuracy achieved with performance measures alone, the addition of physiological measures may well achieve the desired range of accuracy in predicting workload. Models of Mental Workload 2 Mental workload is a multi-faceted phenomenon, and the literature reflects these many facets. Mental workload can be related to physiological states of stress and effort, to subjective experiences of stress, mental effort, and time pressure, and to objective measures of performance levels and to breakdown in performance. These various aspects of workload have led to distinct means for assessing workload including physiological criteria (e.g., heart rate, evoked potentials), performance criteria (e.g., quantity and quality of performance), and subjective criteria (e.g., ratings of level of effort). According to performance criteria, a given task will not necessarily lead to a particular level of performance or workload because factors such as S-R compatibility, practice, fatigue, talent or skill, etc. will affect task workload. For example, a task which may seem overwhelming when first attempted may end up requiring only a small amount of mental capacity after sufficient practice. People learning to fly commonly experience a dramatic reduction in the workload imposed in landing after extended practice. Several aspects of the environment and the aircraft must be monitored and controlled in executing the approach and landing. Particularly at low levels of practice and familiarity, these many aspects of monitoring and control can easily exceed a person’s information processing capacity (but see Schneider & Detweiler, 1988 and Schvaneveldt & Gomez, 1998 for data suggesting that practice on single tasks leads to rather poor transfer to dual task situations). Rogers and Monsell (1995) have shown persistent costs associated with switching between tasks even when the switches are predictable and regular. Schvaneveldt (1969) showed that performance on relatively simple tasks can be degraded when they are coupled with complex, independent tasks. Moray, Dessouky, Kijowski, & Adapathya (1991) showed clear limits to performance in the context of scheduling multiple tasks. Thus, there is reason to believe that the requirement to perform multiple tasks is a major contributor to performance levels and, as a result, to workload (Wickens & Yeh, 1982). The literature on mental workload was extensively reviewed in two chapters in the 1986 Handbook of Perception and Human Performance. On the theoretical side of the problem, understanding workload relates primarily to research in attention, processing capacity, dual-task performance, and allocation of mental resources (Gopher & Donchin, 1986). Assessing workload has involved measurement of performance, subjective impressions of workload, and physiological indicators of work and stress (O’Donnell & Eggemeier; 1986). Because subjective measures of workload have proven useful in a variety of circumstances, we decided to concentrate on these measures in the present investigation. In regard to subjective measures, it is likely that people do not have conscious access to all aspects of mental workload which may cause particular difficulty with subjective measures. Despite this limitation, several studies attest to the value of subjective measures (Bortolussi, Kantowitz, & Hart, 1986; Corwin, 1992; Haskell & Reid, 1987; Reid & Colle, 1988; Tsang & Vidulich, 1994; Vidulich, Ward, & Schueren, 1991; Wierwille & Eggemeier, 1993). It is also important to consider that subjective workload represents the degree to which an individual experiences workload demands, and this experience itself has potential consequences for performance and stress levels. Thus for both theoretical and practical reasons, it is of value to characterize how much mental effort is experienced in performing various tasks and to predict when performance will deteriorate seriously due to overload. Models of Mental Workload 3 Subjective Measures of Mental Workload Subjective measures of mental workload are obtained from subjects’ direct estimates of task difficulty. Various techniques can be used to measure subjective workload, but the basis of any subjective workload technique is having subjects report the “difficulty” of the task. The main difference between subjective workload and other workload measures (such as dual-task or physiological measures of workload) is that the former rely on the subjects’ conscious, perceived experience with regard to the interaction between the operator and the system. Techniques for assessing subjective workload fall either in the category of ratings scales procedures or psychometric techniques involving such procedures as magnitude estimation (e.g. Borg, 1978), paired comparisons (e.g. Wolfe, 1978), or conjoint measurement and scaling (e.g. Reid, Shingledecker, & Eggemeier, 1981). Ratings procedures such as those derived from the Cooper-Harper Aircraft-Handling Characteristics Scale (Cooper & Harper, 1969) require subjects to rate the difficulty of tasks with the use of a decision tree. Ratings scales appear to be sensitive to different levels and varieties of load (including perceptual, central processing, and communications load). An advantage of psychometric, as compared to ratings techniques, is that psychometric techniques are capable of providing interval information regarding task difficulty. Such information can be useful for measuring the magnitude of workload differences between tasks. In magnitude estimation, subjects provide direct estimates of the difficulty of one task relative to another. For example, subjects are exposed to a task and then are told to choose a numerical value reflecting the difficulty associated with the task. The perception of difficulty associated with the first task is called the modulus. Subjects are then asked to provide numerical estimates of tasks of varying difficulty relative to the modulus. Magnitude estimation has proven to be a sensitive measure of differences in load but its major drawback is that real-world tasks often do not occur in close proximity, thus making it difficult for subjects to retain an accurate representation of the modulus over time. In paired-comparisons, subjects are presented with all possible pairs of stimuli (e.g., difficulty levels of a task) and are asked to judge which of the two stimuli are more difficult. After comparisons are obtained from a number of subjects the relative difficulty of stimuli can be represented in an n x n matrix showing the proportion of times each stimulus was judged to be more difficult than every other stimulus. Although this technique has also produced successful results, the number of comparisons required is a limiting factor. For example, with 6 stimuli, 15 judgments are required. An approach which has proven useful is the technique of conjoint measurement and scaling. Most of the techniques used for measuring subjective workload treat perceived workload as a unitary dimension. In some cases subjects are asked to consider multiple factors in making a rating, but subjects still assign one number based on these factors. However, subjective workload entails a number of dimensions, such as time load, mental effort load, and psychological stress load (Reid et al., 1981). Conjoint measurement and scaling approaches are multidimensional techniques with the advantage of reflecting a number of factors in one measure of subjective workload. This approach involves obtaining separate ordinal ratings for each of several dimension of subjective workload and then combining the separate ratings into a scale with interval properties. These techniques require two phases: a scale development phase and an event scoring phase. During the scale development phase levels of dimensions are described to subjects. Then subjects are given all combinations of descriptions of each of the levels for the dimensions and are asked to rank order the combinations according to workload. If there are Models of Mental Workload 4 three dimensions and three levels of difficulty, then subjects would rate 27 combinations. The rankings are then submitted to a series of axiom tests which are part of the conjoint measurement procedure. These axioms are used to test logical consistencies in the data and identify the subject’s combination rule (e.g., additive, distributive, dual distributive) that fits the data. The rule is then used to assign numerical values to each level of the separate dimensions and then combine the values into one integrated scale. During the event-scoring phase, subjects participate in a task and then rate the task difficulty on each of the dimensions. The ratings are then used to find the corresponding value on the subject’s interval scale. Conjoint measures appear to be sensitive to levels of task difficulty. Additional advantages of this approach are that measures are easy to obtain and can be scaled individually to subjects. Subjects are very consistent in their ratings with subjective measures of workload. Reliability coefficients for subjective workload over multiple ratings instances have been as high or higher than .90 (Gopher & Browne, 1984). However, the relationship between subjective and objective measures of workload is variable. In some instances researchers report an association between subjective and objective measures of workload, in other instances dissociations are reported. One explanation for these inconsistencies has to do with the relationship between processes which are and are not available to consciousness. On this view, subjective workload measures will be more sensitive to processes which require awareness (or attention) and less sensitive to processes which do not require attention. According to Gopher and Donchin (1986) the retrospective nature of subjective workload may also be a contributor to the dissociations between subjective and objective measures. Regardless of the limits of subjective measures, the subjective experience of performing a task cannot be ignored. Often, subjective experiences of overload take precedence when an operator is performing a task, even when objective measures are not indicating an overload (Moray, Johanssen, Pew, Rasmussen, Sangers, & Wickens, 1979).